Occupant sensor

ABSTRACT

An occupant sensor incorporates a 3-D imaging system that acquires a 3-D image of an object. The image is segmented to remove unwanted portions and to identify a region-of-interest, and the content thereof is classified responsive to a plurality of 3-D features. In one embodiment, the 3-D image is transformed to a second 3-D image from a second viewing perspective. A two-dimensional projection of the second 3-D image is classified, and a presence, size and position of occupant can be identified from features thereof. A safety restraint system is controlled responsive to the detected scenario, including the presence, position and size of an occupant.

[0001] The instant application claims the benefit of U.S. ProvisionalApplication Ser. No. 60/211,846 filed on Jun. 15, 2000 (5701-00261),which is incorporated herein by reference.

[0002] In the accompanying drawings:

[0003]FIGS. 1a, 1 b and 1 c respectively illustrate front, side and topviews of a three-dimensional (3-D) imaging system in a vehicle;

[0004]FIG. 2 illustrates an arrangement of cameras of a stereo visionsystem;

[0005]FIG. 3 illustrates a model of a stereo imaging process;

[0006]FIG. 4 illustrates a 3-D imaging system using structured lighting;

[0007]FIG. 5 illustrates an image of light stripes by a 3-D imagingsystem of FIG. 4;

[0008]FIG. 6 illustrates a triangulation of a point imaged by a 3-Dimaging system using structured lighting;

[0009]FIG. 7 illustrates a laser scanning system;

[0010]FIG. 8 illustrates a coordinate system of the laser scanningsystem of FIG. 7;

[0011]FIGS. 9a, 9 b, 9 c and 9 d illustrate viewing perspectives fromthe headliner, the driver side, the front, and the top respectively, ofan occupant in the passenger side of a vehicle;

[0012]FIG. 10 illustrates a coordinate system in a vehicle;

[0013]FIG. 11 illustrates an image of a passenger leaning forward,viewed from the headliner;

[0014]FIG. 12 illustrates an image of a passenger leaning forward,viewed from the driver side using coordinate transformations;

[0015]FIG. 13 illustrates an image of a passenger leaning forward,viewed from the front using coordinate transformations;

[0016]FIG. 14 illustrates an image of a passenger leaning forward,viewed from the top using coordinate transformations;

[0017]FIG. 15 illustrates an image of an empty seat, viewed from theheadliner;

[0018]FIG. 16 illustrates an image of an empty seat, viewed from thedriver side using coordinate transformations;

[0019]FIG. 17 illustrates an image of an empty seat, viewed from thefront using coordinate transformations;

[0020]FIG. 18 illustrates an image of an empty seat, viewed from the topusing coordinate transformations;

[0021]FIG. 19 is a flow chart of a process for sensing an occupant andfor controlling a safety restraint system responsive thereto;

[0022]FIG. 20 is a flow chart of a segmentation process;

[0023]FIG. 19 is a flow chart of a classification process;

[0024]FIGS. 22a and 22 b respectively illustrate an uncovered, and acovered, rear facing infant seat located on a passenger seat of avehicle;

[0025]FIG. 23 illustrates a front facing infant seat located on apassenger seat of a vehicle;

[0026]FIG. 24 illustrates a belted occupant seated on a passenger seatof a vehicle;

[0027]FIG. 25 illustrates an occupant reading a newspaper seated on apassenger seat of a vehicle;

[0028]FIGS. 26a, 26 b and 26 c illustrate projections of an empty seat,on the YZ, XZ and XY planes respectively;

[0029]FIGS. 27a, 27 b and 27 c illustrate projections of a rear facinginfant seat, on the YZ, XZ and XY planes respectively;

[0030]FIGS. 28a, 28 b and 28 c illustrate projections of an covered rearfacing infant seat, on the YZ, XZ and XY planes respectively;

[0031]FIGS. 29a, 29 b and 29 c illustrate projections of a front facinginfant, on the YZ, XZ and XY planes respectively;

[0032]FIGS. 30a, 30 b and 30 c illustrate projections of an occupant, onthe YZ, XZ and XY planes respectively;

[0033]FIGS. 31a, 31 b and 31 c illustrate projections of an occupant,reading a newspaper, on the YZ, XZ and XY planes respectively;

[0034]FIG. 32 illustrates an at-risk zone within which an occupant wouldbe out-of-position (OOP) and at risk of injury by the actuation of anassociated air bag inflator module;

[0035]FIG. 33 illustrates a leg occupancy region in front of a seatcushion;

[0036]FIGS. 34a and 34 b illustrate an orientation measure for a rearfacing infant seat (RFIS) and a normally seated occupant, respectively;

[0037]FIGS. 35a and 35 b illustrate a bounding rectangle for a RFIS anda normally seated occupant, respectively;

[0038]FIGS. 36a and 36 b illustrate a best fit ellipse for a RFIS and anormally seated occupant, respectively; and

[0039]FIGS. 37a and 37 b illustrate a central axis line for a RFIS and anormally seated occupant, respectively;

[0040] Referring to FIG. 1, occupant sensor 10 comprises at least oneimaging device 12 in a three-dimensional (3-D) imaging system 14 thatprovides a 3-D image of a scene of a front passenger seat 16 of avehicle. The 3-D image comprises a set of ‘voxels’, or three-dimensionalpixels, each consisting of x, y an z coordinates with respect to arectangular coordinate system.

[0041] The 3-D imaging system 14 can be located at a variety oflocations in view of the seat 16, for example, at the headliner abovethe rear view mirror and pointing towards the passenger seat 16, so asto provide the maximum field of view with minimal obstruction. Thislocation reduces the exposure of the 3-D imaging system 14 to directsunlight and has minimal affect on the appearance of the vehicleinterior.

[0042] However, some locations are less desirable than others. Forexample, if the 3-D imaging system 14 is placed too high on thepassenger side A-pillar it can be obstructed by the sun visor whenpositioned sideways to block the sunlight coming in through the sidewindow. A 3-D imaging system 14 placed low on the A pillar can beobstructed by the occupant's hand(s) or by the occupant reading anewspaper. A 3-D imaging system 14 placed on the dashboard would not‘see’ the whole scene, and would be readily obstructed. The field ofview of a 3-D imaging system 14 placed near the dome light could beobstructed by the head of an occupant. Moreover, such a location wouldnot be desirable for vehicles with sunroofs.

[0043] Various 3-D imaging techniques are capable of providing rangeimages, for example 1) stereo vision, 2) structured lighting and 3)scanning beam (e.g. scanning laser), any of which techniques could beembodied by the 3-D imaging system 14.

[0044] (1) Stereo Vision

[0045] Referring to FIG. 2, a first embodiment of a 3-D imaging system14 is illustrated by a stereo vision system 18 comprising a pair ofsubstantially identical cameras 20.1, 20.2 (e.g. CCD, CMOS or othertechnologies) with substantially identical optics 22 spaced apart by asmall base distance d. The angle 24 between the respective optic axes 26of the cameras is exaggerated in FIG. 2. With the advent of relativelysmall and inexpensive cameras 20.1, 20.2, the stereo vision system 18can be made relatively small. Moreover, these cameras 20.1, 20.2 can beadapted with a logarithmic response to provide a relatively high dynamicrange, so as to prevent or limit saturation when targets are illuminatedby sunlight hits the targets, while at the same time providingsufficient contrast under low ambient lighting conditions, for exampleat night time, perhaps with minimal supplemental infrared (IR)illumination provided by an infrared light emitting diodes (LED) orother illumination source. For example, low power LED's are relativelyinexpensive and safe, and provide illumination that is invisible to thehuman eye—thereby not a distraction—and can be automatically turned onto improve the overall contrast and average intensity of the images, forexample if the overall contrast and average intensity are otherwise low.

[0046] Each camera 20.1, 20.2 captures a respective image 28.1, 28.2 ofthe same scene. Referring to FIG. 3, similar objects in the two imagesare identified by registration thereof with one another, and the 2-DCartesian coordinates (x₁, y₁) and (x₂, y₂) respectively correspondingto a common point of the object are determined from the pixel locationwith respect to the camera coordinate system (x, y). If the worldcoordinate system (X, Y, Z) coincides with that of camera 20.1, then the3-D coordinates (X_(w), Y_(w), Z_(w)) of the target point w are givenby: $\begin{matrix}{Z_{w} = {\lambda - \frac{\lambda \quad d}{x_{2} - x_{1}}}} & (1) \\{X_{w} = \frac{x_{1}\left( {\lambda - Z_{w}} \right)}{\lambda}} & (2) \\{Y_{w} = \frac{y_{1}\left( {\lambda - Z_{w}} \right)}{\lambda}} & (3)\end{matrix}$

[0047] where, λ is the focal length of the lenses of the cameras

[0048] This technique is dependent on the object being imaged havingsufficient detail so as to enable the detection thereof from thecorrelation of the separate stereo images 28.1, 28.2. For the case of alarge area of uniform intensity, for which there is substantially nodetail, in order to prevent the matching process from otherwise failing,a pattern of infrared spots can be projected on the scene (similar tothe structured lighting approach described below), wherein these spotsare used as the reference points that are matched by in the stereoanalysis

[0049] (2) Structured Lighting

[0050] Referring to FIG. 4, a second embodiment of a 3-D imaging system14 comprises a light pattern generator 30 to illuminate a target 32 withstructured lighting 34, and a camera 36 to view the illuminated target32. For example, the camera 36 is a high dynamic response CCD or CMOScamera that is sensitive to both visible and infrared frequencies, andthat is placed at a base distance b from an infrared light patterngenerator 30. The light pattern generator 30, for example, comprises aninfrared laser source with a stripe generator that projects a lightpattern 38 of multiple parallel lines or stripes on the target 32. Thecamera 36 captures an image of the target 32, upon which is superimposedthe light pattern. The signal to noise ratio of the imaged light pattern38 can be improved by strobing the light pattern 38 at half thefrequency of the frame rate of the camera 36 so that alternate imageshave the light pattern 38 superimposed on the image of the target 32,and the remaining images do not. Subtracting an image frame without asuperimposed light pattern 38 from an adjacent image frame with thesuperimposed light pattern provides a resultant image—for a stationarybackground—of substantially only the light pattern 38, as illustrated inFIG. 5. The light pattern 38 can be made brighter than sunlight, evenwith a relatively lower power density, because the light pattern 38 isstrobed and the whole scene can be illuminated for a relatively brieftime interval with relatively bright light from the light patterngenerator 30. Accordingly, the subtraction process for extracting thelight pattern 38 can be done under arbitrary lighting conditions withoutcompromising occupant safety.

[0051] The spacing of the lines 40 of the light pattern 38 superimposedon the target 32 depends on the distance of the target from the 3-Dimaging system 14, and the distortion thereof depends on the shape ofthe target 32. The actual 3-D coordinates are measured usingtriangulation of the light spots that constitute the light pattern 38.In FIG. 6, the coordinate system (x,y) of the camera 36 is coincidentwith the world coordinate system (X, Y, Z); the base separation betweenthe light source and the camera 36 is b and the light source lies on theX axis, i.e. the light source center is at (b,0,0); the Z axis is theoptical axis of the camera 36; the focal length of the camera lens is f;so that the image plane lies at Z=f. The exemplary generated lightpattern 38 comprises a series of parallel lines 40, for example, Nparallel lines 40, wherein each line 40 comprises a collection of lightpoints, for example, M light points on each line 40 (as determined bythe resolution of the camera 36). Each line 40 results from theprojection of an associated light plane on the target 32. For the k^(th)light plane (generating the k^(th) line) subtending an angle γ_(k) withthe ZX plane (k=1, 2, . . . N), the projection of the line joining thecenter of the light source to the q^(th) point of the k^(th) line ontothe ZX plane is at angle α_(kq) with respect to the X axis (q=1, 2, . .. , M ). If the point P corresponding to the q^(th) point on the k^(th)line is imaged at the point p(x, y) on the image, the world coordinatesof P: (X₀, Y₀, Z₀) are given by: $\begin{matrix}{X_{0} = \frac{b\quad x\quad \tan \quad \alpha_{kq}}{f + {x\quad \tan \quad \alpha_{kq}}}} & (4) \\{Y_{0} = \frac{b\quad y\quad \tan \quad \alpha_{kq}}{f + {x\quad \tan \quad \alpha_{kq}}}} & (5) \\{Z_{0} = \frac{b\quad f\quad \tan \quad \alpha_{kq}}{f + {x\quad \tan \quad \alpha_{kq}}}} & (6)\end{matrix}$

[0052] The coordinates are independent of γ_(k), the angle made by thek^(th) light plane with the ZX plane.

[0053] (3) Scanning Laser

[0054] Referring to FIG. 7, a third embodiment of a 3-D imaging system14—a scanning laser range finder 42—comprises a scanning laser 44 thatscans a laser beam spot 46 across the target 32 in accordance with araster scan pattern. The range to each point is triangulated by anoptical ranging sensor 48, e.g. a photo sensitive detector. Referring toFIG. 8, the 3-D coordinates of the target point P are determined inspherical coordinates (R,α,θ), where R is the range from the sensor, αand θ are the azimuth and elevation angles respectively. The azimuth andelevation angles are known from the azimuth and elevation resolutionrespectively of the scanning system, which for example scans in equalincrements. The rectangular coordinates (X₀, Y₀, Z₀) of the target pointP are related to the spherical coordinates as follows:

X ₀ =R·cos θ·sin α  (7)

Y ₀ =R·sin θ  (8)

Z ₀ =R·cos θ·cos α  (9)

[0055] Data Analysis

[0056] Regardless of the 3-D imaging technique, the 3-D imaging system14 provides a set of 3-D coordinates of the scene. Referring to FIG. 19,the resulting 3-D data is used in an occupant sensing process that canbe used for controlling the actuation of a safety restraint system. Withthe 3-D imaging system 14 installed in the vehicle, the location—i.e.the orientation and position—of the coordinate systems of the camera(s)and the world coordinate system are fixed. The 3-D coordinates of apoint on a target 32 can be expressed with respect to any worldcoordinate system at any position and orientation using coordinatetransformations. In other words, the 3-D image taken from the fixedlocation at the headliner can be effectively viewed from any otherlocation of choice (for example, from the headliner, either of theA-pillars, the dashboard, the driver side or other locations) by usingone or more coordinate transformations.

[0057] As an example, FIGS. 9a-d illustrate a laboratory setup of avehicle interior buck viewed from four different perspectives asfollows: from headliner (FIG. 9a), from the driver side (FIG. 9b), fromthe front (FIG. 9c) and from the top (FIG. 9d). Referring to FIG. 10,for the coordinate system origin at the headliner above the rear viewmirror as illustrated in FIG. 9a, the positive x axis is horizontal andtowards the driver side, the positive y axis is vertical and towards thefloor and the positive z axis is horizontal and towards the back of thevehicle. 3-D image data, respectively of an occupant seated leaningslightly forward and the empty seat, was collected from this locationusing an infrared scanning laser range finder 42. The respective imagesfrom the headliner location are shown in FIGS. 11 and 15 respectively.These same images are respectively transformed to the viewingperspectives of the driver side, the front and the top by transformationof coordinate systems, as shown in FIGS. 12 through 14 respectively forthe occupant seated leaning slightly forward, and in FIGS. 16 through 18respectively for the empty seat.

[0058] Segmentation of the Scene

[0059] As used herein, the term segmentation means the extraction fromthe image of a region of interest (ROI) that contains usefulinformation. Referring to FIGS. 19 and 20, the side door, A-pillar,dashboard, floor and objects outside the window are all examples ofbackground clutter that can be and preferably are eliminated from theimage by segmentation, leaving as a remainder the ROI. This reduces thenumber of data points that need to be processed by a recognitionalgorithm.

[0060] The dashboard, side door and the floor can be characterized asfixed planes. For example, the plane representing the side door can becharacterized as:

g·x+h·y+I·z=n  (10)

[0061] With the door closed—as would be the case with the vehicle inmotion—this plane is fixed and g, h, i and n are fixed parameters of thevehicle. The points on the door are eliminated by comparing a linearcombination of the data points (X, Y, Z) with a threshold, as follows:

g·X+h·Y+i·Z−n<T ₀ (threshold)  (11)

[0062] wherein those points satisfying equation (11) are sufficientlyclose to the fixed plane to be assumed to be associated with the door.

[0063] Similar calculations are done for the dashboard and the floor toeliminate the visible portions of these features. The A-pillar ischaracterized by a fixed curved surface, the parameters of which dependon the particular vehicle:

f(x, y, z)=s  (12)

[0064] If the function f(x, y, z) cannot be expressed in a standardform, then the function can, for example, be characterized by a leastsquares fit of a functional form, using the actual 3-D coordinates ofthe A-pillar. The same process can be used in modeling a dashboard of anonstandard shape. The visible portion of the A-pillar, and othervisible features such as the dashboard that are similarly characterized,are eliminated from the image using the criterion:

f(X, Y, Z)−s<T ₁ (threshold)  (13)

[0065] Points outside the side window—for example, having large negativex coordinates—are discarded by comparing with a threshold T₂corresponding to the distance from the origin of the coordinate systemto the side door plane that is roughly parallel to the YZ plane.Therefore, the point (X Y, Z) is outside if:

X<−T ₂  (14)

[0066] If the image is not of an empty seat (method of detecting emptyseats is described below), then the portion of the empty seat that isvisible is also be segmented out.

[0067] Classification of Scenarios

[0068] Referring to FIGS. 19 and 21, following the segmentation of theimage, the image is analyzed to determine whether or not the seat isempty. For an empty seat, the image comprises a seat cushion (bottom)and a seat back, which can be respectively characterized by tworespective planes—a first plane characterizing the seat cushion and asecond plane, at an angle relative to the first, characterizing the seatback.

[0069] An equation of a seat back plane, for the seat back completelyreclined and the seat cushion fully forward and horizontal, is given by:

d·x+e·y+f·z=m  (15)

[0070] wherein the parameters d, e, f and m are fixed for a particularvehicle. The angle of the seatback and the position and recline of theseat cushion are all variable, so the equation of the seat back plane isa function of these three factors. Referring to FIG. 10, the seatcushion travels principally along the Z axis. Moreover, the seat backrotates about an axis that is substantially parallel to the seat baseand to the X axis, which is also substantially parallel and close to aroughly straight line given by the intersection of the seat back and theseat cushion planes. The equation of the seat back plane, for a givenposition and slope of the seat cushion and a given recline of the seatback, are determined by first applying a translational transformationmatrix T that provides a translation along the Z axis, and then applyinga rotational transformation matrix R_(α) to account for the rotationwith respect to the X axis. If Δz and Δα represent a significant changein the seat cushion travel and the seat back angle, then any giventranslation z of the seat cushion from the front-most position, and anygiven rotation angle α of the seat back from the complete reclineposition, can be represented by multiples of Δz and Δα respectively,wherein Δz and Δα are parameters of the particular vehicle.

[0071] More particularly, the equation of the seatback plane for a giventranslation z of the seat cushion and recline α of the seat back isdetermined from the following operations:

[0072] The translational transformation matrix T is given by:$\begin{matrix}{T = \begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & z \\0 & 0 & 0 & 1\end{bmatrix}} & (16)\end{matrix}$

[0073] The rotational transformation matrix R_(α) is given by:$\begin{matrix}{R_{\alpha} = \begin{bmatrix}1 & 0 & 0 & 0 \\0 & {\cos \quad \alpha} & {\sin \quad \alpha} & 0 \\0 & {{- \sin}\quad \alpha} & {\cos \quad \alpha} & 0 \\0 & 0 & 0 & 1\end{bmatrix}} & (17)\end{matrix}$

[0074] The new coordinates (x′, y′, z′) are determined from the oldcoordinates (x, y, z) by $\begin{matrix}{\begin{bmatrix}x^{\prime} \\y^{\prime} \\z^{\prime} \\1\end{bmatrix} = {{\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & z \\0 & 0 & 0 & 1\end{bmatrix}\quad\begin{bmatrix}1 & 0 & 0 & 0 \\0 & {\cos \quad \alpha} & {\sin \quad \alpha} & 0 \\0 & {{- \sin}\quad \alpha} & {\cos \quad \alpha} & 0 \\0 & 0 & 0 & 1\end{bmatrix}}\quad {\quad{\left\lbrack {\quad{\quad\begin{matrix}x \\y \\z \\1\end{matrix}}} \right\rbrack \quad = \begin{bmatrix}x \\{{y\quad \cos \quad \alpha} + {z\quad \sin \quad \alpha}} \\{{{- y}\quad \sin \quad \alpha} + {z\quad \cos \quad \alpha} + z} \\1\end{bmatrix}}}}} & (18)\end{matrix}$

[0075] The equation of a plane characterizing the seat back is givenfrom equation (18) by:

d _(α,z) ·x+e _(α,z) ·y+f _(60 ,z) ·z=m _(α,z)  (19)

[0076] where,

d _(α,z) =d  (20)

e _(α,z) =e·cos α−f·sin α  (21)

f _(α,z) =e·sin α+f·cos α+f  (22)

m _(α,z) =m  (23)

[0077] A seat cushion at a horizontal tilt and an arbitrarytranslational position is characterized by the plane:

a·x+b·y+c·z=k  (24)

[0078] wherein the parameters a, b, c and k are fixed for a particularvehicle. The equation of the plane for any other tilt of the seatcushion is found by applying the above described rotationaltransformation about the X axis.

[0079] Clusters of points lying on the seat cushion plane of equation(24) and seat back plane of equation (19) are checked to see if theyform the rough shape of the seat cushion and back respectively, bychecking test points (X, Y, Z) to see if the following equations aresatisfied:

a·X+b·Y+c·Z−k<T ₃ (threshold)  (25)

d _(α,z) ·X+e _(α,z) ·Y+f _(α,z) ·Z−m _(α,z) <T ₄ (threshold)  (26)

[0080] for all possible combinations of seat cushion position and seatcushion slope and seat back angle.

[0081] If a seat bottom is not detected, the seat is assumed occupied,wherein the possible seat occupancy scenarios are for example forwardfacing infant or child seat (FFIS), RFIS or an occupant. This is donegenerally from the volumetric shape of the region of interest. The seatback may or may not be visible, and visible portions of the seat aresegmented out of the image.

[0082] Once the scene is identified in a ‘macro’ level, individual partsof the scene are identified. For example, the image is then searched tofind a somewhat spherical shape representing a head. Referring to FIG.11, the image of the target has a dominant spherical region. The searchbegins by finding a roughly spherical surface satisfying the equation(x−a_(h))²+(y−b_(h))²+(z−c_(h))²=r_(h) ², where (a_(h), b_(h), c_(h)) isthe centroid of the spherical region and r_(h) is the radius. The searchbegins with a reasonable guess as to where the head is likely to be in3-D space for the particular vehicle, after which the position of thecenter of the sphere, and the radius of the sphere, are respectivelyiterated by the search.

[0083] The image is then searched to find cylindrical surfacesrepresenting the arms and legs. The torso, is characterized by arelatively flat surface. Semantics are used—a spherical surface (head)with two cylindrical surfaces (arms) on both sides, a relatively lesscurved surface below the spherical surface (torso) and in between thetwo cylindrical surfaces (arms), the cylindrical surfaces originatingfrom the top of the less curved surface, two more cylindrical surfaces(legs) originating from the bottom of the less curved surface—allindicate an occupant. The size of these features can be roughlydetermined to distinguish the size of the occupant, e.g. large, mediumor small.

[0084] If the seat is occupied and none of the above are observed, thelikely candidate is a RFIS. Referring to FIGS. 22a and 22 b, a RFIS maybe uncovered or covered. A substantial portion of the seat back isvisible for either of these cases, but more so with the uncovered RFIS.A ‘kidney bean’ shape is indicative of the uncovered RFIS, in which casetwo small cylindrical surfaces maybe visible on the right representingthe legs of the infant. A somewhat smooth surface is indicative of acovered RFIS.

[0085] Referring to FIG. 23, an occupant in a FFIS or booster seat isindicated if all of the above limbs are visible and they are relativelysmall, and if the occupant is not seated directly on the seat, but issomewhat raised thereabove, as indicated by an outer boundary of theoccupant zone that is not completely planar. A child in a booster seatis indicated if the seatback is visible but the occupant is seated on araised surface, as determined by looking at the buttocks region to seehow far it is from the seat cushion plane.

[0086] Referring to FIG. 24, seatbelt usage may also be determined fromsurface characteristic, for example, the presence of a somewhatelongated and arched surface.

[0087] Referring to FIG. 25, an occupant reading a newspaper isidentified by looking for a large planar surface on the left of thescene and likely a spherical surface because the head may be seen fromover the newspaper.

[0088] Aside from the modeling shapes of the surfaces, mathematicalfeatures are also used for robust classification of features, whereinshape descriptors are applied to the 3-D segmented ROI for volumetricanalysis. Furthermore, the projections of the volume on the XY, YZ, andZX planes—respectively corresponding to the front, side and top views ofthe ROI volume respectively shown in FIGS. 13, 12 and 14—are analyzed in2-D. Most of the individual features cannot alone distinguish betweenscenarios, but may individually distinguish between certain propertiesof the scenarios. Accordingly, all the features are combined in afeature vector that is formed for an overall classification.

[0089] 3-D Features

[0090] The 3-D features are given, for example, as follows:

[0091] (1) Volumetric Central Moments: Central moments are shapedescriptors independent of the position of the ROI. The central momentof order p, q, r (p,q,r=0, 1, 2 . . . ) is defined by: $\begin{matrix}{\mu_{pqr} = {\sum\limits_{x}{\sum\limits_{y}{\sum\limits_{z}{\left( {x - \overset{\_}{x}} \right)^{p}\left( {y - \overset{\_}{y}} \right)^{q}\left( {z - \overset{\_}{z}} \right)^{r}}}}}} & (27)\end{matrix}$

[0092] where ({overscore (x)}, {overscore (y)}, {overscore (z)}) is thecentroid of the ROI from equations (29-31). The moment of order p, q, ris defined by: $\begin{matrix}{m_{pqr} = {\sum\limits_{x}{\sum\limits_{y}{\sum\limits_{z}{x^{p}y^{q}z^{r}}}}}} & (28)\end{matrix}$

[0093] Moments are essentially shape descriptors. However they aredependent on the spatial position of the object. Equation (27) providesfor spatial invariance so that the moment values will be the same forsimilar ROI's regardless of their corresponding location in the vehicle.For example, the central moments of a RFIS would be the same for anyposition of the vehicle seat.

[0094] (2) Centroids $\begin{matrix}{\overset{\_}{x} = \frac{m_{100}}{m_{000}}} & (29) \\{\overset{\_}{y} = \frac{m_{010}}{m_{000}}} & (30) \\{\overset{\_}{z} = \frac{m_{001}}{m_{000}}} & (31)\end{matrix}$

[0095] Centroids provide a position in 3-D space that can be a usefulindicator of the seat occupancy scenario. For example, referring to FIG.10, a RFIS would be closer to the instrument panel, thus having a lower{overscore (z)} value, than would a normally seated occupant having ahigher {overscore (z)} value. The {overscore (x)} value provides thelateral position of the target, thus providing an indication if anoccupant is seated in the middle of a bench seat. The {overscore (y)}centroid enables tall objects to be distinguished from short objects—aRFIS tends to be lower thus having a lower {overscore (y)} value ascompared to that of a normally seated occupant.

[0096] (3) Volume

V=m ₀₀₀  (32)

[0097] Occupants, child seats and empty seats typically have differentvolumes. This feature is especially useful in determining the size ofthe occupant, once the image has been classified.

[0098] (4) Volumetric Roundness: This is a measure of the roundness ofthe ROI ranging from 0 to 1, where 1 corresponds to a perfectlyspherical ROI, as given by:

R _(v)=6·π² ·V/p _(v) ³  (33)

[0099] where, V is the volume and p_(v) is the average of the perimetersof the projections of the ROI on the XY, YZ and ZX planes. Child seatstend to be more ‘spherical’ than people. Moreover, the empty seat has adifferent roundness.

[0100] (5) Ratio of Radii. A radius is a line segment joining thecentroid to any point on the outer boundary of the ROI. The ratio of themaximum (R_(max)) and minimum (R_(min)) radii is a feature, as given by:

r _(R) =R _(max) /R _(min)  (34)

[0101] This measure is roughly analogous to aspect ratio—‘thinner’objects, for example occupants and empty seats, typically have a highervalue than ‘compact’ objects, for example child seats.

[0102] (6) Volume of the Bounding Cube: The geometric mean of the areasof the bounding rectangles for the three projections of equation (56) isthe volume of the bounding cube, as given by:

V _(B) ={square root}{square root over (A_(Bxy)A_(Byz)A_(Bzx))}  (35)

[0103] where,

[0104] A_(Bxy)=Area of the rectangle bounding the XY projection of the3-D ROI;

[0105] A_(Byx)=Area of the rectangle bounding the YZ projection of the3-D ROI; and

[0106] A_(Bzx)=Area of the rectangle bounding the ZX projection of the3-D ROI.

[0107] This is another way of analyzing the volume of the target.

[0108] (7) Ratio of Volumes: This is the ratio of the actual volume V tothat of the bounding cube V_(B), as given by:

R _(v) =V/V _(B)  (36)

[0109] Targets with large portions sticking out from the main body, forexample an occupant with stretched arms, will have a large V_(B)compared to its volume V since a large portion of the boundingrectangles typically contain more than the projections of the ROI. Childseats, which generally do not have large objects jutting out therefrom,typically are characterized by a value of R_(V) close to 1, whereasoccupants with hands extended or legs on the dashboard would have a muchlower value of R_(V).

[0110] (8) Percentage Volume Occupied: Referring to FIG. 30, the regionin front of the seat cushion known as the Leg Occupancy Region is likelyto be occupied by the legs of the occupant and is likely to be empty forRFIS, FFIS and empty seats. Thus the ratio of the portion of the volume(V_(o)) of the ROI occupying this region to the volume V_(p) of theregion is likely to be high for occupants and low for RFIS, FFIS andempty seats. This ratio is given by:

V _(p) =V _(o) /V _(p)  (37)

[0111] 2-D Features

[0112] Referring to FIGS. 26 through 31, the 2-D features calculated onthe three projections of the ROI provide substantial shape information.These 2-D features are illustrated hereinbelow for the projection on theXY plane. The corresponding features for the projections on the YZ andZX planes are determined by replacing (x, y) by (y, z) and (z, x)respectively:

[0113] (1) Central Moments: Central moments are position independentshape descriptors, and are given by: $\begin{matrix}{\mu_{pq} = {\sum\limits_{x}{\sum\limits_{y}{\left( {x - \overset{\_}{x}} \right)^{p}\left( {y - \overset{\_}{y}} \right)^{q}}}}} & (39)\end{matrix}$

[0114] wherein the centroids are given by: $\begin{matrix}{\overset{\_}{x} = \frac{m_{10}}{m_{00}}} & (40) \\{\overset{\_}{y} = \frac{m_{01}}{m_{00}}} & (41)\end{matrix}$

[0115] (2) Normalized central moments: These shape descriptors arerotation, scale and translation independent, and are given by:$\begin{matrix}{{\eta_{pq} = \frac{\mu_{pq}}{\mu_{00}^{\gamma}}}{{w\quad h\quad e\quad r\quad e},}} & (41) \\{\gamma = {\frac{p + q}{2} + 1}} & (42)\end{matrix}$

[0116] (3) Invariant Moments: These scale, rotation and translationinvariant moments are robust shape descriptors (Digital ImageProcessing, Gonzalez, Woods), and are given by:

φ₁=η₂₀+η₀₂  (43)

φ₂=(η₂₀−η₀₂)²+4η₁₁ ²  (44)

φ₃=(η₃₀−3η₁₂)²+(3η₂₁−η₀₃)²  (45)

φ₄=(η₃₀+η₁₂)²+(η₂₁+η₀₃)²  (46) $\begin{matrix}{\varphi_{5} = {{\left( {\eta_{30} - {3\eta_{12}}} \right){\left( {\eta_{30} + \eta_{12}} \right)\left\lbrack {\left( {\eta_{30} + \eta_{12}} \right)^{2} - {3\left( {\eta_{21} + \eta_{03}} \right)^{2}}} \right\rbrack}} + {\left( {{3\eta_{21}} - \eta_{03}} \right){\left( {\eta_{21} + \eta_{03}} \right)\left\lbrack {{3\left( {\eta_{30} + \eta_{12}} \right)^{2}} - \left( {\eta_{21} + n_{03}} \right)^{2}} \right\rbrack}}}} & (47)\end{matrix}$

 φ₆=(η₂₀−η₀₂)[(η₃₀+η₁₂)²−(η₂₁+η₀₃)² ]+4η ₁₁(η₃₀+η₁₂)(η₂₁+η₀₃)  (48)$\begin{matrix}{\varphi_{7} = {{\left( {{3\eta_{21}} - \eta_{30}} \right){\left( {\eta_{30} + \eta_{12}} \right)\left\lbrack {\left( {\eta_{30} + \eta_{12}} \right)^{2} - {3\left( {\eta_{21} + \eta_{03}} \right)^{2}}} \right\rbrack}} + {\left( {{3\eta_{12}} - \eta_{30}} \right){\left( {\eta_{21} + \eta_{03}} \right)\left\lbrack {{3\left( {\eta_{30} + \eta_{12}} \right)^{2}} - \left( {\eta_{21} + n_{03}} \right)^{2}} \right\rbrack}}}} & (49)\end{matrix}$

[0117] (4) Perimeter: The perimeter is a measure of the size of the ROI,and is given by: $\begin{matrix}{p = {\sum\limits_{i = 1}^{M}\sqrt{\left( {x_{i} - x_{i - 1}} \right)^{2} + \left( {y_{i} + y_{i - 1}} \right)^{2}}}} & (50)\end{matrix}$

[0118] The perimeter of the projection of an empty seat is likely to beless than that of an occupied seat.

[0119] (5) Area:

A=m ₀₀  (51)

[0120] The area of the projection of an empty seat is likely to be lessthan that of a RFIS, FFIS or occupant.

[0121] (6) Roundness: The roundness of the projection is 1 for perfectcircles and less than 1 for other shapes, and is given by:

R=4πA/p ²  (52)

[0122] A RFIS would have a different measure of roundness than anoccupant or an empty seat.

[0123] (7) Bending Energy: The bending energy is a measure of the curvesin the shape of the projections (Fundamentals of Digital ImageProcessing, Anil K. Jain), and is given by:

[0124] Where $\begin{matrix}{E_{b} = \left. {\frac{1}{p}\overset{p}{\int\limits_{0}}} \middle| {k(t)} \middle| {}_{2}{t} \right.} & (53) \\{{k(t)} = \sqrt{\left( \frac{^{2}x}{t^{2}} \right)^{2} + \left( \frac{^{2}y}{t^{2}} \right)^{2}}} & (54)\end{matrix}$

[0125] and t is the distance along the perimeter from any arbitrarystarting point on the perimeter. The bending energy is high for shapeswith many sharp bends as would result for occupants. Child seats wouldtend to have a lower value of bending energy.

[0126] (8) Orientation: Referring to FIG. 34, this is a measure of theangle the projection makes with the independent axis, and is given by:$\begin{matrix}{\theta = {\frac{1}{2}{\tan^{- 1}\left( \frac{2\quad \mu_{11}}{\mu_{20} - \mu_{02}} \right)}}} & (55)\end{matrix}$

[0127] This feature is relatively strong for the projection on the YZplane since the RFIS would be tilted leftwards, as illustrated in FIG.27a, thus having a small orientation angle versus that of a normallyseated occupant, illustrated in FIG. 30a, or a FFIS, illustrated in FIG.29a.

[0128] (9) Area of the Bounding Rectangle: Referring to FIG. 35, this isthe smallest rectangle enclosing the projection after it is rotatedabout its orientation angle, and is given by:

A _(b) =L _(b) ·W _(b)  (56)

[0129] where, first the projection is rotated by θ (the orientation):

α=x·cos θ+y·sin θ  (57)

β=−x·sin θ+y·cos θ  (58)

[0130] and then the length (L_(b)) and width (W_(b)) of the rectangleare determined from:

L _(b)=(α_(max)−α_(min)  (59)

W _(b)=β_(max)−β_(min)  (60)

[0131] This measure is typically different for different images.

[0132] (10) Best Fit Ellipse: Referring to FIG. 36, the best fit ellipseis given by (x/a)²+(y/b)²=1, where the associated features are given by:$\begin{matrix}{{{Semi}\quad {major}\quad {axis}} = {a = {\left( \frac{4}{\pi} \right)^{1/4}\left\lbrack \frac{I_{\max}^{3}}{I_{\min}} \right\rbrack}^{1/8}}} & (61) \\{{{Semi}\quad {minor}\quad {axis}} = {b = {\left( \frac{4}{\pi} \right)^{1/4}\left\lbrack \frac{I_{\min}^{3}}{I_{\max}} \right\rbrack}^{1/8}}} & (62)\end{matrix}$

[0133] where, $\begin{matrix}\begin{matrix}{I_{\max} = {{Greatest}\quad {moment}{\quad \quad}{of}{\quad \quad}{inertia}}} \\{= {\sum{\sum\limits_{}\left\lbrack {{\left( {y - \overset{\_}{y}} \right)\sin \quad \theta} + {\left( {x - \overset{\_}{x}} \right)\cos \quad \theta}} \right\rbrack^{2}}}}\end{matrix} & (63) \\\begin{matrix}{I_{\min} = {{Least}\quad {moment}{\quad \quad}{of}{\quad \quad}{inertia}}} \\{= {\sum{\sum\limits_{}\left\lbrack {{\left( {y - \overset{\_}{y}} \right)\cos \quad \theta} + {\left( {x - \overset{\_}{x}} \right)\sin \quad \theta}} \right\rbrack^{2}}}}\end{matrix} & (64)\end{matrix}$

[0134]

is the region consisting of the projection.

[0135] The following are also features obtained from the best fitellipse:

Area of the ellipse=A _(ellilpse) =π·a·b  (65)

[0136] $\begin{matrix}{{{Volume}{\quad \quad}{rendered}{\quad \quad}{by}\quad {the}\quad {ellipse}} = {V_{ellipse} = {\pi \frac{4a^{2}b}{3}}}} & (66) \\{{{Eccentricity}\quad {of}\quad {the}\quad {ellipse}} = {E_{ellipse} = \sqrt{1 - \left( \frac{b}{a} \right)^{2}}}} & (67)\end{matrix}$

 Eccentric center of the ellipse=C _(ellipse) =a·e  (68)

Eccentric normal=N _(ellipse)=2b ² /a  (69)

[0137] Occupants are more ‘elongated’ than child seats especially whenviewed from the driver side. Accordingly, the ellipse bounding themwould typically be substantially different from an ellipse bounding achild seat. Stated another way, the features describing the ellipse foran occupant are typically different from those for child seats and emptyseats.

[0138] (11) Eccentricity of the ROI Projection: This is a measure of theelongation, and is given by: $\begin{matrix}{e_{{p\quad r\quad {oj}}\quad} = \frac{\left( {\mu_{20} - \mu_{02}} \right)^{2} + {4\mu_{11}}}{A_{b\quad l\quad o\quad b}}} & (70)\end{matrix}$

[0139] Occupants typically have a larger eccentricity than those ofchild seats and empty seats because occupants are typically moreelongated.

[0140] (12) Ratio of Areas: This measure is given by the ratio of thearea of the blob to the area of the bounding rectangle, as follows:

R _(a) =A/A _(b)  (71)

[0141] This measure is relatively small for regions with largeprotruding parts. e.g., occupants with arms extended.

[0142] (13) Central Axis Line: The projection is rotated by theorientation angle θ to a 0° orientation angle, after which straightlines are drawn vertically through the projection. A 2^(nd) order fit ofthe mid points of the portions of these lines bounded by the perimeteris rotated back to its original orientation, resulting in:

f(x)=a ₀ +a ₁ ·x+a ₂ ·x ²  (72)

[0143] Referring to FIG. 37, the central axis lines for a RFIS and anormally seated occupant typically have different curvatures.Accordingly, the coefficients a₁ and a₂ are features that indicate thecurvature of the central axis line.

[0144] After the elements of the test feature vector f are calculated,as given by:

f=[f ₁ f ₂ f ₃ . . . f _(n)]^(T)  (73)

[0145] the test feature vector is compared with reference (or “golden”)feature vectors for the various scenarios f_(s), where s is thescenario, for example s⊂{RFIS, FFIS, Occupant, Empty Seat}

f _(s) =[f _(s1) f _(s2) f _(s3) . . . f _(sn)]^(T)  (74)

[0146] by comparing the vector distance d_(s)

[0147] $\begin{matrix}{d_{s} = \sqrt{\sum\limits_{i = 1}^{n}\left( {f_{i} - f_{si}} \right)^{2}}} & (75)\end{matrix}$

[0148] The classification is done, for example, using a minimum distanceclassifier, whereby the detected scene is the one for which thecorresponding golden feature vector is nearest (d_(s) is minimum) to thetest feature vector.

[0149] OOP Occupant Detection

[0150] The distance of the identified scene or portions of the scenefrom the instrument panel is then identified by looking at thecoordinates from a perspective perpendicular to the length of thevehicle. Therefore, it can be determined whether the identified targetis within an “at-risk” zone, regardless of shape or size. The lateralposition of the occupant/object can also be determined using the 3-Dcoordinates. Once the image is identified, the position of parts of theimage are tracked from frame to frame by assigning a tag thereto, afterobserving that no change in the initial scene occurs from frame to frameand observing the relative displacements of the individual components.Accordingly, the position of the identified parts of the occupant isfound in 3-D space, which aids in identifying an out of position (OOP)occupant, regardless of the size and shape of the “at-risk” zone andregardless of the definition of an OOP occupant (e.g. whether or nothands inside the “at-risk” zone constitutes an OOP occupant), which isuseful for situations with dynamic “at-risk” zones.

[0151] Determination of the Size of the Occupant and Restraint Control

[0152] The 3-D data also provides a rough estimate of the volume, andaccordingly the size of the occupant—if present,—which information canbe used to control the deployment of the airbag. The decision for thedeployment of the airbag or the type of deployment can be determined,for example, as follows: the air bag would be turned off for RFIS oroccupants at certain postures deemed at risk from the airbag (out ofposition (OOP) occupant), the deployment may be softer for a smalleroccupant closer to the dashboard.

[0153] The occupant sensor 10 can be used on the driver side by imagingthe driver, for example from the same headliner location as used toimage the passenger, in order to determine the size of the driver, andthe position of the torso, head and arms, any of which can be used totrack the driver's movement over time, in order to tailor the deploymentof the airbag.

[0154] The 3-D imaging system 14 acquires range images, which differfrom 2-D images in that the pixel values represent distances from theimaging system, as opposed to intensity. By obtaining a range image ofx, y, z points, the scene can be viewed from any perspective bytranslating and/or rotating the coordinate axes. The segmentationprocess becomes easier and more robust as the background clutter outsidethe window can be eliminated since their position in 3-D space is known.Similarly the fixed objects (dashboard, door etc) in view can beeliminated since they have fixed coordinates. With 3-D coordinates, theshape descriptors contain more separable information to enhance theclassification of the scene—these give an idea of the 3-D volume versusthe 2-D shape. Finally, the position of each data point can be clearlydetermined with respect to any part of the vehicle thus enabling thedetection of an out of position (OOP) occupant, which is defined as somepart of the occupant within some predefined “at-risk” zone. With a 3-Dsystem, an OOP occupant can be determined for an “at-risk” zone ofarbitrary shape or size. Looking at the sequence of range images,arbitrary points can be tracked over time thus enabling the tracking ofthe occupant during pre-crash or even crash periods. Using 3-Dcoordinates the approximate volume and hence the size of the target canbe determined.

[0155] The above-described 3-D imaging system 14 incorporates an imageprocessor and associated electronics for acquiring and processing theassociated imaging data. The safety restraint system is controlledresponsive to the above-described processing of the imaging data, eitherby the image processor, or by a separate control processor. Generally,the safety restraint system is actuated responsive to a crash asdetected by a crash sensor, provided that the actuation thereof isenabled responsive to the above-described image processing by the imageprocessor.

[0156] While specific embodiments have been described in detail in theforegoing detailed description and illustrated in the accompanyingdrawings, those with ordinary skill in the art will appreciate thatvarious modifications and alternatives to those details could bedeveloped in light of the overall teachings of the disclosure.Accordingly, the particular arrangements disclosed are meant to beillustrative only and not limiting as to the scope of the invention,which is to be given the full breadth of the appended claims and any andall equivalents thereof.

I claim:
 1. A method of sensing an occupant in a vehicle, comprising: a.providing for acquiring a first three-dimensional image of a scene froma first viewing perspective; b. providing for segmenting said firstthree-dimensional image so as to identify a region-of-interest in saidfirst three-dimensional image; c. providing for forming a secondthree-dimensional image by removing a portion of said firstthree-dimensional image that is outside of said region-of-interest; andd. providing for classifying a scenario responsive to an image contentof said second three-dimensional image, wherein said image content isrepresented by a plurality of three-dimensional features selected from avolumetric central moment, a centroid, a volume, a volumetric roundness,a ratio of radii, a volume of a bounding cube, a ratio of volumes and apercentage of volume occupied.
 2. A method of sensing an occupant in avehicle as recited in claim 1, wherein said portion of said firstthree-dimensional image that is outside of said region-of-interestcomprises a portion of an image of an object selected from a dashboardof the vehicle, an interior of a side door of a vehicle, a scene outsidea window of the vehicle, a floor of the vehicle, and a structural pillarin the vehicle;
 3. A method of sensing an occupant in a vehicle asrecited in claim 1, wherein said operation of classifying comprisesdetecting the presence of an occupant from a plurality of said featuresof said second three-dimensional image.
 4. A method of sensing anoccupant in a vehicle as recited in claim 3, further comprisingproviding for tracking said occupant from one image frame to another. 5.A method of sensing an occupant in a vehicle as recited in claim 3,further comprising providing for detecting whether said occupant islocated in an at-risk zone proximate to a safety restraint system.
 6. Amethod of sensing an occupant in a vehicle as recited in claim 3,further comprising providing for determining a size of said occupantfrom at least one feature of said second three-dimensional image.
 7. Amethod of sensing an occupant in a vehicle as recited in claim 1,further comprising providing for controlling a safety restraint systemresponsive to said operation of classifying a scenario.
 8. A method ofsensing an occupant in a vehicle, comprising: a. providing for acquiringa first three-dimensional image of a scene from a first viewingperspective; b. providing for transforming said first three-dimensionalimage to a second three-dimensional image from a second viewingperspective; c. providing for segmenting either said firstthree-dimensional image prior to said transforming operation, or saidsecond three-dimensional image, so as to identify a region-of-interestin said first or said second three-dimensional image; d. providing forforming a third three-dimensional image by removing a portion of saidfirst or said second three-dimensional image that is outside of saidregion-of-interest; and e. providing for classifying a scenarioresponsive to an image content of said third three-dimensional image,wherein said image content comprises a two-dimensional representation ofsaid third three-dimensional image.
 9. A method of sensing an occupantin a vehicle as recited in claim 8, wherein said portion of said firstor said second three-dimensional image that is outside of saidregion-of-interest comprises a portion of an image of an object selectedfrom a dashboard of the vehicle, an interior of a side door of avehicle, a scene outside a window of the vehicle, a floor of thevehicle, and a structural pillar in the vehicle.
 10. A method of sensingan occupant in a vehicle as recited in claim 8, wherein said imagecontent is represented by at least one two-dimensional feature selectedfrom a central moment, a normalized central moment, an invariant moment,a perimeter, and area, a roundness, a bending energy, an orientation, anarea of a bounding rectangle, a bet fit ellipses, and eccentricity of aregion of interest, a ratio of areas, and a central axis line.
 11. Amethod of sensing an occupant in a vehicle as recited in claim 10,wherein said operation of classifying comprises detecting the presenceof an occupant from a plurality of said features of said thirdthree-dimensional image.
 12. A method of sensing an occupant in avehicle as recited in claim 11, further comprising providing fortracking said occupant from one image frame to another.
 13. A method ofsensing an occupant in a vehicle as recited in claim 11, furthercomprising providing for detecting whether said occupant is located inan at-risk zone proximate to a safety restraint system.
 14. A method ofsensing an occupant in a vehicle as recited in claim 11, furthercomprising providing for determining a size of said occupant from atleast one feature of said third three-dimensional image.
 15. A method ofsensing an occupant in a vehicle as recited in claim 8, furthercomprising providing for controlling a safety restraint systemresponsive to said operation of classifying a scenario.
 16. A method ofsensing an occupant in a vehicle, comprising: a. providing for acquiringa first three-dimensional image of a scene from a first viewingperspective; b. providing for segmenting said first three-dimensionalimage so as to identify a region-of-interest in said firstthree-dimensional image; c. providing for forming a secondthree-dimensional image by removing a portion of said firstthree-dimensional image that is outside of said region-of-interest; d.providing for generating a first representation of a plurality of pointswithin said region-of-interest; e. comparing said representation with aplurality of a priori representations, wherein each a priorirepresentation represents an object to be identified at a particularposition and orientation, and if a difference between said firstrepresentation and one of said a priori representations is less than athreshold, then indicating that first three-dimensional image containsan image of said object.